Accelerated Backpropagation Learning : Parallel Tangent Optimization Algorithm

Accelerated Backpropagation Learning: Extended Dynamic Parallel Tangent Optimization Algorithm

A Novel Fast Backpropagation Learning Algorithm Using Parallel Tangent and Heuristic Line Search

In gradient based learning algorithms, the momentum has usually an improving effect in convergence rate and decreasing the zigzagging phenomena. However it sometimes causes the convergence rate to decrease. The Parallel Tangent (ParTan) gradient is used as deflecting method to improve the convergence. From the implementation point of view, it is as simple as the momentum. In fact this method is...

Backpropagation training in adaptive quantum networks

We introduce a robust, error-tolerant adaptive training algorithm for generalized learning paradigms in high-dimensional superposed quantum networks, or adaptive quantum networks. The formalized procedure applies standard backpropagation training across a coherent ensemble of discrete topological configurations of individual neural networks, each of which is formally merged into appropriate lin...

Accelerated Backpropagation Learning: Two Optimization Methods

Two methods for incr easing performance of th e backpropagat ion learning algorithm are present ed and their result s are compared with those obtained by optimi zing par ameters in the standard method . The first method requires adaptation of a scalar learning rat e in order to decrease th e energy value along the gradient direction in a close-to-optimal way. Th e second is derived from the con...

Revisit Long Short-Term Memory: An Optimization Perspective

Long Short-Term Memory (LSTM) is a deep recurrent neural network architecture with high computational complexity. Contrary to the standard practice to train LSTM online with stochastic gradient descent (SGD) methods, we propose a matrix-based batch learning method for LSTM with full Backpropagation Through Time (BPTT). We further solve the state drifting issues as well as improving the overall ...